System and Method for Controlling a Display of a Mobile Device

ABSTRACT

A method and system are provided for controlling the display of an mobile device by: capturing an image using a camera device of the mobile device, the camera device being directed in a same direction as a display of the mobile device, the image comprising one or more subjects (e.g. users or other humans seen in the image); determining a point of regard in the image for at least one of the one or more subjects, the point of regard being indicative of an area on the display at which a gaze of the corresponding subject is directed; determining, based on the point of regard, an instruction for controlling the display; and controlling the display according to the instruction, wherein controlling the display includes reducing visibility of at least one portion of what is displayed on the display.

CROSS-REFERENCE TO RELATED APPLICATIONS

The present application is a continuation of U.S. patent applicationSer. No. 12/938,799 filed on Nov. 3, 2010 incorporated herein byreference.

TECHNICAL FIELD

The following relates generally to controlling a display of a mobiledevice.

DESCRIPTION OF THE RELATED ART

Many mobile devices include a display to provide a medium for the mobiledevice to present information to a user. Such mobile devices having adisplay can automatically dim or turn off its display after a predefinedtime-out period. For example, the time-out period can be defined as acertain length of time of inactivity of the mobile device's inputs (e.g.keyboard, number pad, buttons, touchscreen, trackball, trackpad, etc.).However, with a predefined time-out period, the display in somecircumstances may be turned off while the user is still viewing thedisplay. Such circumstances can occur, for example, if the usercontinues using the mobile device for a length of time that exceeds thetime-out period without activating or otherwise using the inputs (e.g.while reading an e-mail or watching a video). On the other hand, thedisplay may be left on after the user has stopped using the device for alength of time equal to the remaining portion (if any) of the time-outperiod in order to activate the automatic shut-off of the display.

BRIEF DESCRIPTION OF THE DRAWINGS

Embodiments will now be described by way of example only with referenceto the appended drawings wherein:

FIG. 1 is a schematic diagram of a user viewing the display of a mobiledevice.

FIG. 2 is a plan view of an example mobile device and a display screentherefor.

FIG. 3 is a plan view of another example mobile device and a displayscreen therefor.

FIG. 4 is a block diagram of an example embodiment of a mobile device.

FIG. 5 is a screen shot of a home screen displayed by the mobile device.

FIG. 6 is a block diagram illustrating example ones of the othersoftware applications and components shown in FIG. 5.

FIG. 7 is a block diagram of an example configuration of an eye trackingdisplay controller application.

FIG. 8 is a flow diagram of example computer executable instructions forcontrolling a display based on a user's point of regard.

FIG. 9 is a flow diagram of an example process for determining a lengthof time to track a user of a mobile device.

FIG. 10 is a flow diagram of an example process for determining a pointof regard by eye tracking.

FIG. 11 is a flow diagram of an example process for approximating apoint of regard by face tracking.

FIG. 12 is a schematic diagram of two users viewing the display of amobile device.

FIG. 13 is a flow diagram of example computer executable instructionsfor controlling a display based on the point of regard of one or moreusers.

FIG. 14 is a flow diagram of example computer executable instructionsfor calibrating a display controller module.

FIG. 15 is an example screen shot illustrating a portion of a screen ofa mobile device being obscured with another portion unchanged.

DETAILED DESCRIPTION

It will be appreciated that for simplicity and clarity of illustration,where considered appropriate, reference numerals may be repeated amongthe figures to indicate corresponding or analogous elements. Inaddition, numerous specific details are set forth in order to provide athorough understanding of the example embodiments described herein.However, it will be understood by those of ordinary skill in the artthat the example embodiments described herein may be practised withoutthese specific details. In other instances, well-known methods,procedures and components have not been described in detail so as not toobscure the example embodiments described herein. Also, the descriptionis not to be considered as limiting the scope of the example embodimentsdescribed herein.

It has been recognized that methods for automatically controlling adisplay of an electronic device such as a mobile device are typicallylimited in their ability to determine when the device is being used andwhen it is idle. To address this, the following describes a method ofcontrolling the display of an electronic device by: capturing an imageusing a camera device of the electronic device, the camera device beingdirected in a same direction as a display of the electronic device, theimage comprising one or more subjects (e.g. users or other humans seenin the image); determining a point of regard in the image for at leastone of the one or more subjects, the point of regard being indicative ofan area on the display at which a gaze of the corresponding subject isdirected; determining, based on the point of regard, an instruction forcontrolling the display; and controlling the display according to theinstruction, wherein controlling the display includes reducingvisibility of at least one portion of what is displayed on the display.

In some example embodiments, the method includes determining under whatone or more criteria to track the one or more subjects using the cameradevice and capturing the image upon meeting the one or more criteria.The one or more criteria may be associated with any one or more of thefollowing: a period time, a location associated with the electronicdevice, content being displayed on the display, and a mode of theelectronic device.

The method may also include obtaining status information on theelectronic device, and capturing the image based on the statusinformation. The status information may include any one or more of thefollowing: change in orientation, interactivity with one or more inputdevices, one or more applications currently being used, and a mode ofthe electronic device. The status information may also be used todetermine under what one or more criteria to track the one or moresubjects using the camera device and capturing the image upon meetingthe one or more criteria as noted above.

The image may be captured upon determining that the electronic device isin a security mode and the instruction for controlling the display maycorresponds to a manner in which to modify what is viewed on thedisplay. The display can be modified by performing any one or more ofthe following: turning off the display, initiating a screensaver,dimming a backlight, adjusting a focus of content being displayed, andlocking-out the device. In addition, one or more portions of what isbeing displayed can be modified such that the visibility of those one ormore portions is/are reduced whereas the other portions (e.g. clock,ribbon, etc.) are displayed as they normally would. Controlling thedisplay according to the principles discussed herein can be advantageousfor security purposes, to conserve battery power, or both.

It can be appreciated that determining the point of regard can beperformed using any one or both of eye tracking and face tracking.

Turning to FIG. 1, a schematic diagram of a user 2 viewing a display 12of a mobile device 100 is provided. In this example, the mobile device100 is situated in front of the user 2. The user 2 has a pair of eyes 4that have associated therewith, a gaze direction 6 (i.e. the directiontowards which the user is looking), and a point of regard 8 (i.e. anarea at which the user is looking). In this example, the gaze direction6 is towards the display 12, and the point of regard 8 falls on aviewing area 13 provided thereby. The mobile device 100 also has a frontor forward-facing camera lens 30, a light source 32, and a distancesensor 34, collectively referred to as a camera device 31. The frontcamera lens 30, light source 32, and distance sensor 34 can be used totrack the gaze direction 6 and point of regard 8 of the user 2. Thepoint of regard 8 can be used to determine and thus control the display12 in order to reduce the visibility of what is being displayed. Forexample, the mobile device 100 may be operable such that when the pointof regard 8 falls within the viewing area 13, the display 12 is on (e.g.by illuminating a backlight), and when the point of regard 8 fallsoutside the viewing area 13, the display 12 is turned off. In otherexample embodiments described below, the presence of multiple points ofregard 8, in particular when viewing sensitive data, can cause thevisibility of the contents to be reduced (e.g. by turning off thescreen) for security purposes.

It can therefore be seen that the display 12 can be effectivelycontrolled by detecting that the user is looking at or away from theviewing area 13 of the display 12, or by detecting that a second user islooking at the viewing area 13. As will be discussed later on, themobile device 100 can be configured to control the display 12 in variousways when one or more points of regard 8 are detected as being inside oroutside the viewing area 13.

Examples of applicable electronic devices include pagers, cellularphones, cellular smart-phones, wireless organizers, personal digitalassistants, computers, laptops, handheld wireless communication devices,wirelessly enabled notebook computers, camera devices, tablet computers,and the like. Such devices will hereinafter be commonly referred to as“mobile devices” for the sake of clarity. It will however be appreciatedthat the principles described herein are also suitable to other devices,e.g. “non-mobile” devices.

In an example embodiment, the mobile device can be a two-waycommunication device with advanced data communication capabilitiesincluding the capability to communicate with other mobile devices orcomputer systems through a network of transceiver stations. The mobiledevice may also have the capability to allow voice communication.Depending on the functionality provided by the mobile device, it may bereferred to as a data messaging device, a two-way pager, a cellulartelephone with data messaging capabilities, a wireless Internetappliance, or a data communication device (with or without telephonycapabilities).

Referring to FIGS. 2 and 3, one example embodiment of a mobile device100 a is shown in FIG. 2, and another example embodiment of a mobiledevice 100 b is shown in FIG. 3. It will be appreciated that the numeral“100” will hereinafter refer to any mobile device 100, including theexample embodiments 100 a and 100 b, those example embodimentsenumerated above or otherwise. It will also be appreciated that asimilar numbering convention may be used for other general featurescommon between all figures such as a display 12, a positioning device14, a cancel or escape button 16, a camera button 17, and a menu oroption button 24.

The mobile device 100 a shown in FIG. 2 includes a display 12 a with aviewing area 13 a and the cursor or view positioning device 14 shown inthis example embodiment is a trackball 14 a. Positioning device 14 mayserve as another input member and is both rotational to provideselection inputs to the main processor 102 (see FIG. 4) and can also bepressed in a direction generally toward housing to provide anotherselection input to the processor 102. Trackball 14 a permitsmulti-directional positioning of the selection cursor 18 (see FIG. 5)such that the selection cursor 18 can be moved in an upward direction,in a downward direction and, if desired and/or permitted, in anydiagonal direction. The trackball 14 a is in this example situated onthe front face of a housing for mobile device 100 a as shown in FIG. 2to enable a user to manoeuvre the trackball 14 a while holding themobile device 100 a in one hand. The trackball 14 a may serve as anotherinput member (in addition to a directional or positioning member) toprovide selection inputs to the processor 102 and can preferably bepressed in a direction towards the housing of the mobile device 100 b toprovide such a selection input. It can be appreciated that the trackball14 a is only one example of a suitable positioning device 14. Forexample, a trackpad, touchscreen, OLED, or other input mechanism mayequally apply.

The display 12 may include a selection cursor 18 that depicts generallywhere the next input or selection will be received. The selection cursor18 may includee a box, alteration of an icon or any combination offeatures that enable the user to identify the currently chosen icon oritem. The mobile device 100 a in FIG. 2 also includes a programmableconvenience button 15 a to activate a selected application such as, forexample, a calendar or calculator. Further, mobile device 100 a includesan escape or cancel button 16 a, a camera button 17 a, a menu or optionbutton 24 a and a keyboard 20 a. The camera button 17 a is able toactivate photo and video capturing functions, e.g. when pressed in adirection towards the housing. The menu or option button 24 a can beused to load a menu or list of options on the display 12 a when pressed.In this example, the escape or cancel button 16 a, the menu optionbutton 24 a, and a keyboard 20 a are disposed on the front face of themobile device housing, while the convenience button 15 a and camerabutton 17 a are disposed at the side of the housing. This buttonplacement enables a user to operate these buttons while holding themobile device 100 in one hand. The keyboard 20 a is, in this exampleembodiment, a standard QWERTY keyboard, however, it will be appreciatedthat reduced QWERTY or virtual keyboards (e.g. as provided by atouchscreen) may equally apply

The mobile device 100 a also has a front camera lens 30 a, a lightsource 32 a and a distance sensor 34 a, collectively referred to as acamera device 31 a. The light source 32 a may be used to illuminate anobject (e.g. user 2) for capturing an image such as a photo, or acollection of images such as a video. The front camera lens 32 a allowsthe light that represents an image to enter into the camera device 31 a.The camera device 31 a may be activated by pressing the camera button 17a,. The distance sensor 34 a measures or determines the distance betweenthe front camera lens 32 a and an object in the image captured by thecamera device 31 a.

The mobile device 100 b shown in FIG. 3 includes a touch-screen display12 b with a viewing area 13 b and the positioning device 14 in thisexample embodiment is a trackpad 14 b. The mobile device 100 b alsoincludes a menu or option button 24 b, a cancel or escape button 16 b, acamera button 17 b, a convenience button 15 b, a front camera lens 30 b,a light source 32 b and a distance sensor 34 b. The front camera lens 30b, light source 32 b and distance sensor 34 b are collectively referredto as a camera device 31 b. The mobile device 100 b, as illustrated inFIG. 3, includes a standard reduced QWERTY keyboard 20 b. In thisexample embodiment, the keyboard 20 b, positioning device 14 b, escapebutton 16 b and menu button 24 b are disposed on a front face of amobile device housing.

It will be appreciated that for the mobile device 100, a wide range ofone or more positioning or cursor/view positioning mechanisms such as atouch/track pad, a positioning wheel, a joystick button, a mouse, atouchscreen, a set of arrow keys, a tablet, an accelerometer (forsensing orientation and/or movements of the mobile device 100 etc.),OLED, or other whether presently known or unknown may be employed.Similarly, any variation of keyboard 20 may be used. It will also beappreciated that the mobile devices 100 shown in FIGS. 2 and 3 are forillustrative purposes only and various other mobile devices 100 areequally applicable to the following examples. Other buttons may also bedisposed on the mobile device housing such as colour coded “Answer” and“Ignore” buttons to be used in telephonic communications. In anotherexample, the display 12 may itself be touch sensitive thus itselfproviding an input mechanism in addition to display capabilities.

To aid the reader in understanding the structure of the mobile device100, reference will now be made to FIGS. 4 through 6.

Referring first to FIG. 4, shown therein is a block diagram of anexample embodiment of a mobile device 100. The mobile device 100includes a number of components such as a main processor 102 thatcontrols the overall operation of the mobile device 100. Communicationfunctions, including data and voice communications, are performedthrough a communication subsystem 104. The communication subsystem 104receives messages from and sends messages to a wireless network 200. Inthis example embodiment of the mobile device 100, the communicationsubsystem 104 is configured in accordance with the Global System forMobile Communication (GSM) and General Packet Radio Services (GPRS)standards, which is used worldwide. Other communication configurationsthat are equally applicable are the 3G and 4G networks such as EDGE,UMTS and HSDPA, LTE, Wi-Max etc. New standards are still being defined,but it is believed that they will have similarities to the networkbehaviour described herein, and it will also be understood by personsskilled in the art that the example embodiments described herein areintended to use any other suitable standards that are developed in thefuture. The wireless link connecting the communication subsystem 104with the wireless network 200 represents one or more different RadioFrequency (RF) channels, operating according to defined protocolsspecified for GSM/GPRS communications.

The main processor 102 also interacts with additional subsystems such asa Random Access Memory (RAM) 106, a flash memory 108, a display 110, anauxiliary input/output (I/O) subsystem 112, a data port 114, a keyboard116, a speaker 118, a microphone 120, a GPS receiver 121, short-rangecommunications 122, a camera 123, a accelerometer 125, a distance sensor127 and other device subsystems 124. The display 110 can be atouch-screen display able to receive inputs through a user's touch.

Some of the subsystems of the mobile device 100 performcommunication-related functions, whereas other subsystems may provide“resident” or on-device functions. By way of example, the display 110and the keyboard 116 may be used for both communication-relatedfunctions, such as entering a text message for transmission over thenetwork 200, and device-resident functions such as a calculator or tasklist.

The mobile device 100 can send and receive communication signals overthe wireless network 200 after required network registration oractivation procedures have been completed. Network access is associatedwith a subscriber or user of the mobile device 100. To identify asubscriber, the mobile device 100 may use a subscriber module componentor “smart card” 126, such as a Subscriber Identity Module (SIM), aRemovable User Identity Module (RUIM) and a Universal SubscriberIdentity Module (USIM). In the example shown, a SIM/RUIM/USIM 126 is tobe inserted into a SIM/RUIM/USIM interface 128 in order to communicatewith a network. Without the component 126, the mobile device 100 is notfully operational for communication with the wireless network 200. Oncethe SIM/RUIM/USIM 126 is inserted into the SIM/RUIM/USIM interface 128,it is coupled to the main processor 102.

The mobile device 100 is typically a battery-powered device and includesa battery interface 132 for receiving one or more rechargeable batteries130. In at least some example embodiments, the battery 130 can be asmart battery with an embedded microprocessor. The battery interface 132is coupled to a regulator (not shown), which assists the battery 130 inproviding power to the mobile device 100. Although current technologymakes use of a battery, future technologies such as micro fuel cells mayprovide the power to the mobile device 100.

The mobile device 100 also includes an operating system 134 and softwarecomponents 136 to 146 which are described in more detail below. Theoperating system 134 and the software components 136 to 146 that areexecuted by the main processor 102 are typically stored in a persistentstore such as the flash memory 108, which may alternatively be aread-only memory (ROM) or similar storage element (not shown). Thoseskilled in the art will appreciate that portions of the operating system134 and the software components 136 to 146, such as specific deviceapplications, or parts thereof, may be temporarily loaded into avolatile store such as the RAM 106. Other software components can alsobe included, as is well known to those skilled in the art.

The subset of software applications 136 that control basic deviceoperations, including data and voice communication applications, may beinstalled on the mobile device 100 during its manufacture. Softwareapplications may include a message application 138, a device statemodule 140, a Personal Information Manager (PIM) 142, a connect module144 and an IT policy module 146. A message application 138 can be anysuitable software program that allows a user of the mobile device 100 tosend and receive electronic messages, wherein messages are typicallystored in the flash memory 108 of the mobile device 100. A device statemodule 140 provides persistence, i.e. the device state module 140ensures that important device data is stored in persistent memory, suchas the flash memory 108, so that the data is not lost when the mobiledevice 100 is turned off or loses power. A PIM 142 includesfunctionality for organizing and managing data items of interest to theuser, such as, but not limited to, e-mail, contacts, calendar events,and voice mails, and may interact with the wireless network 200. Aconnect module 144 implements the communication protocols that arerequired for the mobile device 100 to communicate with the wirelessinfrastructure and any host system, such as an enterprise system, thatthe mobile device 100 is authorized to interface with. An IT policymodule 146 receives IT policy data that encodes the IT policy, and maybe responsible for organizing and securing rules such as the “SetMaximum Password Attempts” IT policy.

Other types of software applications or components 139 can also beinstalled on the mobile device 100. These software applications 139 canbe pre-installed applications (i.e. other than message application 138)or third party applications, which are added after the manufacture ofthe mobile device 100. Examples of third party applications includegames, calculators, utilities, etc.

The additional applications 139 can be loaded onto the mobile device 100through at least one of the wireless network 200, the auxiliary I/Osubsystem 112, the data port 114, the short-range communicationssubsystem 122, or any other suitable device subsystem 124.

The data port 114 can be any suitable port that enables datacommunication between the mobile device 100 and another computingdevice. The data port 114 can be a serial or a parallel port. In someinstances, the data port 114 can be a USB port that includes data linesfor data transfer and a supply line that can provide a charging currentto charge the battery 130 of the mobile device 100.

For voice communications, received signals are output to the speaker118, and signals for transmission are generated by the microphone 120.Although voice or audio signal output is accomplished primarily throughthe speaker 118, the display 110 can also be used to provide additionalinformation such as the identity of a calling party, duration of a voicecall, or other voice call related information.

Turning now to FIG. 5, the mobile device 100 may display a home screen40, which can be set as the active screen when the mobile device 100 ispowered up and may constitute the main ribbon application. The homescreen 40 in this example generally includes a status region 44 and atheme background 46, which provides a graphical background for thedisplay 12. The theme background 46 displays a series of icons 42 in apredefined arrangement on a graphical background. In some themes, thehome screen 40 may limit the number icons 42 shown on the home screen 40so as to not detract from the theme background 46, particularly wherethe background 46 is chosen for aesthetic reasons. The theme background46 shown in FIG. 5 provides a grid of icons. It will be appreciated thattypically several themes are available for the user to select and thatany applicable arrangement may be used. An example icon shown in FIG. 5is a camera icon 51 used to indicate a camera-based application such asa typical camera or video application for capturing images and video,or, as will be described below, a camera-based eye tracking displaycontroller application 60 (see also FIG. 6). One or more of the seriesof icons 42 is typically a folder 52 that itself is capable oforganizing any number of applications therewithin, as well as or insteadof images, videos, data files, etc.

The status region 44 in this example embodiment includes a date/timedisplay 48. The theme background 46, in addition to a graphicalbackground and the series of icons 42, also includes a status bar 50.The status bar 50 provides information to the user based on the locationof the selection cursor 18, e.g. by displaying a name for the icon 53that is currently highlighted.

An application, such as message application 138 may be initiated (openedor viewed) from display 12 by highlighting a corresponding icon 53 usingthe positioning device 14 and providing a suitable user input to themobile device 100. For example, message application 138 may be initiatedby moving the positioning device 14 such that the icon 53 is highlightedby the selection box 18 as shown in FIG. 5, and providing a selectioninput, e.g. by pressing the trackball 14 a.

FIG. 6 shows an example of other software applications and components139 that may be stored and used on the mobile device 100. Only examplesare shown in FIG. 6 and such examples are not to be consideredexhaustive. In this example, an alarm application 54 may be used toactivate an alarm at a time and date determined by the user. There isalso an address book 62 that manages and displays contact information. AGPS application 56 may be used to determine the location of a mobiledevice 100. A calendar application 58 that may be used to organizeappointments. Another example application is a display controllerapplication 60. As will be discussed below, the display controllerapplication 60 may be operable to control the display 12 of a mobiledevice 100 based on the point of regard 8 of a user 2.

Turning to FIG. 7, an example configuration of the display controllerapplication 60 is provided. The display controller application 60 canrequest details of activity occurring on, or receive inputs from, inputdevices such as the keyboard 116 and the accelerometer 125, in order todetermine an estimate of interactivity with the mobile device 100. Insome example embodiments, the keyboard 116 and accelerometer 125 sendinformation regarding their respective statuses (e.g. activityinformation) to the display controller application 60. In other exampleembodiments the display controller application 60 polls the keyboard andaccelerometer 125 to determine interactivity. It can be appreciated thatthe use of interactivity with the keyboard 116 and accelerometer 125 areonly two examples and other input mechanisms such as touch-screenelements, buttons, etc. are also applicable. The display controllerapplication 60 in this example also receives inputs from the distancesensor 127 and the camera 123 in this example. By obtaining such inputs,the camera 123 can obtain an image of the object (e.g. a user) as viewedor otherwise captured by the front camera lens 30, and then may send theimage to the display controller application 60. The distance sensor 127in this example is operable to obtain or calculate the distance betweenthe front camera lens 30 and the object viewed by the front camera lens30, and then send the information to the display controller application60.

The display controller application 60 in this example also includes acommence tracking module 210 for determining when to instruct the camera123 to take an image, which is communicable with the distance sensor 127to determine the distance between the front camera lens 30 and theobject in the image. The display controller application 60 alsoincludes, in this example, a point of regard module 212 for determiningthe point of regard of the user 2 in an image, a point of regard storage214 for storing point of regard data, a calibration module 211 forcalibrating the display controller application 60, and a displaycontroller 216 for sending instructions to the display 110 to modify theproperties of the display 12 (e.g. power, brightness and contentdisplayed), in order to reduce visibility of what is being displayed.

The commence tracking module 210 obtains information about the status ofthe mobile device 100 from various components and device subsystems (seeFIG. 4) of the mobile device 100. In the example shown in FIG. 7, theeye tracking display controller application 60 obtains statusinformation (e.g. activity information) from the keyboard 116 andaccelerometer 125 device subsystems. For example, using statusinformation from the keyboard 116 and accelerometer 125, the commencetracking module 60 is able to determine whether to instruct the camera123 to capture an image and the distance sensor 127 to take or otherwiseobtain a reading.

The point of regard module 212 obtains an image from the camera 123 andthe distance between the front camera lens 30 and the object captured inthe image from the distance sensor 127. Using the image, the point ofregard module 212 is able to detect a user 2 or subject, if one can befound in the captured image. Using the image and the distance reading,the point of regard module 212 is then able to calculate the point ofregard 8 of the user 2 or subject in the image. The point of regardinformation is stored in the point of regard storage 214.

The calibration module 211 can be used to enable the display controllerapplication 60 to train itself for detecting subjects in the image andhow they look in the image and what readings of the accelerometercorrespond thereto. For example, the display controller application 60can initiate a training process to have the user 2 look at the device indifferent positions (e.g. held to the left, right, above, below, etc.)in order to be able to calibrate how to detect the point of regard 8. Itmay be noted that during the calibration process, the display controllerapplication 60 can be operable to prompt the user to both look straightat the display 12 for a particular held position, and then look awayfrom the mobile device 10 at the same position. In this way, the displaycontroller application 60 is provided with two points of reference tothereafter judge whether or not the user is likely glancing at thedisplay 12.

The display controller module 216 retrieves the point of regardinformation from the point of regard storage 214 and uses thisinformation to select instructions to send to the display 110.

It will be appreciated that any module or component exemplified hereinthat executes instructions or operations may include or otherwise haveaccess to computer readable media such as storage media, computerstorage media, or data storage devices (removable and/or non-removable)such as, for example, magnetic disks, optical disks, or tape. Computerstorage media may include volatile and non-volatile, removable andnon-removable media implemented in any method or technology for storageof information, such as computer readable instructions, data structures,program modules, or other data, except transitory propagating signalsper se. Examples of computer storage media include RAM, ROM, EEPROM,flash memory or other memory technology, CD-ROM, digital versatile disks(DVD) or other optical storage, magnetic cassettes, magnetic tape,magnetic disk storage or other magnetic storage devices, or any othermedium which can be used to store the desired information and which canbe accessed by an application, module, or both. Any such computerstorage media may be part of the mobile device 100 or accessible orconnectable thereto. Any application or module herein described may beimplemented using computer readable/executable instructions oroperations that may be stored or otherwise held by such computerreadable media.

Turning to FIG. 8, example computer executable instructions are providedfor controlling the display 12 of a mobile device based on eye tracking.At block 300, status information on the mobile device 100 is received orotherwise obtained. For example, the status information can include,without limitation, a change in orientation of the mobile device 100,interactivity with one or more input devices, one or more applicationscurrently being used, and a mode (e.g. security mode) of the mobiledevice 10. Based on the status information, under what criteria to trackthe user 2 of the mobile device 100 is determined at block 302. Forexample, a period of time, specified area or location, use of aparticular application, presence of a plurality of subjects in a fieldof view of the camera device, mode of the device, etc. can be used todetermine how long or under what circumstances to track the user 2. Atblock 304, an image of the user 2 of the mobile device 100 is capturedusing the front camera lens 30 of the mobile device 100. Using the imagecaptured in block 304, the point of regard 8 of the user 2 as capturedin the image is determined at block 306. At block 308, a displaycontroller instruction is selected based on the point of regard 8 of theuser 2. For example, the instruction may correspond to a manner in whichto modify what is viewed on the display 12 to reduce the visibilitythereof (e.g. turning off the display 12, initiating a screensaver,dimming a backlight, adjusting a focus of content being displayed,locking-out the device, etc.) The instruction may also correspond to amanner in which one or more portions of what is being displayed is/aremodified to reduce visibility thereof whilst leaving the other portionsto be displayed normally. It has been recognized that in addition to orrather than reducing the visibility of everything on the display 12,where the display 12 is capable of, for example, dimming one portionwhile leaving another portion as is, part of the display 12 can beobscured or concealed without affecting other, typically less sensitiveportions of the display 12 such as the clock or ribbon. Such an exampleis shown in FIG. 15 wherein a clock and date/time information portion800 is unaffected while the remaining portion 802 of the screen 13 isobscured or blurred.

As noted above, status information on the mobile device 100 is receivedor otherwise obtained at block 300. In the example configuration of thedisplay controller application 60 shown in FIG. 7, the statusinformation may include the keyboard activity and the accelerometeractivity. It can be appreciated that the status information can includeother information on the mobile device 100 such as the activity of otherinputs, the current state of the display and user profile settings. Thenature of the status information can depend on the criteria used inblock 302 to determine a time to track the user 2.

As also noted above, at block 302, under what criteria to track the user2 is determined. In the example configuration of FIG. 7, block 302 maybe implemented by the commence tracking module 210. An example of aprocess to determine a time to track the user is provided in FIG. 9. InFIG. 9, the accelerometer 125 provides the status information of themobile device 100 or the commence tracking module 210 requests statusinformation. At block 314, the accelerometer 125 status information,thus obtained, is evaluated for activity. If the accelerometer 125status information indicates activity (e.g. the mobile device 100 isbeing picked up or put away), block 314 determines it is time to trackthe user 2. The relative activity according to the accelerometer 125 cansuggest that the mobile device 100 is going to be used (e.g. the mobiledevice 100 is picked up) or the mobile device 100 is no longer beingused (e.g. the mobile device 100 is being put away). In either case, thedisplay 12 can be instructed to perform a change in state from off to onor vice versa and, as such an accelerometer 125 can be used to providean indication of a change in use of the display 12.

It can be appreciated that the accelerometer status information can beused to improve the efficiency of the display controller application 60.For example, by taking readings from the accelerometer 125 whiletracking a user 2, changes in orientation can be detected between imagesthat would indicate that the process could be aborted or that a changein how the subject appears in the image should be expected. This allowsthe display controller application 60 to make the process faster whentaking the next image. For example, in cases where the mobile device 100is flipped or otherwise rotated quickly, the accelerometer 125 caneasily detect that this motion corresponds to a situation wherein theuser is likely not looking at the mobile device 100. That is because insuch a case, for the user to be looking at the mobile device 100 theywould need to be also turning or flipping themselves through the samemotion, which is unlikely. As a result, the display controllerapplication 60 can detect that eye-tracking is not applicable at thattime and can instruct the camera 123 to not take the next image (or failto instruct the camera 123 to take the next image) at that time.

In another example embodiment, rather than or in addition to, the statusinformation is provided by or requested from the keyboard 116. If thekeyboard 116 is in use, the user is more likely to be viewing thedisplay 12 and there may therefore be no need to track the user 2.However, inactivity in the keyboard 116 can conversely suggest that themobile device 100 is not in use and the user 2 should be tracked toconfirm this.

In yet another example embodiment, the status information can be basedon the type of content that is being displayed. Based on the contenttype, block 302 can increase or decrease the frequency in which the user2 is tracked. For example, the tracking frequency can be proportional tothe sensitivity of the content being displayed. The display of privateand confidential e-mail could increase the tracking frequency so thatthe display 12 is immediately turned off once the user 2 is not viewingthe display 12 to minimize the opportunity for others to view thecontent. In contrast, a music playlist would be an example of displaycontent that would require less frequent tracking of the user 2.

It is appreciated that other information about the mobile device 100 caninclude status information for determining a time to track the user 2. Acombination of information may also be used (e.g. accelerator, keyboardand content type together).

At block 304, an image of the user in the example embodiments shownherein is captured using the front camera lens 30 of the mobile device100. In one example embodiment, a light source 32 adjacent to the frontcamera lens 30 is used to illuminate an object when captured by thefront camera lens 30 and, as result, capture an image of what is beingilluminated by the light source 32, e.g. the user 2. It can beappreciated that the light source 32 can be of any suitable type,including visible light, infrared light, etc.

As noted above, at block 306, the point of regard of the user 2 capturedin the image taken by front camera lens 30 is determined. Examplemethods of implementing block 306 are now provided. In one exampleembodiment, block 306 can be implemented by using eye trackingalgorithms or analytics (i.e. a routine or set of instructions fortracking the gaze direction 6 of the user's eyes 4), in order tocalculate the point of regard 8.

By way of background, the concept of eye tracking is well known and anumber of different techniques have been developed for accomplishing eyetracking. A commonly used technique is based on so-called pupil centercorneal reflection (PCCR) methods. PCCR methods typically involve usinga camera device 31 to capture an image of a user 2 while the user 2 isbeing illuminated by the light source 32. It has been found that PCCRmethods may work optimally when using an infrared light source 32,however, the principles may be applied using visible light sources 32.The light source 32 illuminates the user's eyes 4 and the illuminationcauses highly visible reflections of the light in parts of the user'seyes 4. A fraction of the light enters the pupil and is reflected offthe retina to identify the pupils of the eyes 4. A fraction of the lightis reflected off the cornea of the eyes 4, the reflection often referredto as a “glint”. Using known image processing techniques, the centers ofthe pupil and the glint are identified and a vector from the center ofthe glint to the center of the pupil is calculated. The direction ofthis vector provides gaze direction information which is used with thedistance between the camera device 31 and the user's eyes 4 to calculatethe point of regard 8 using known methods.

In another example embodiment, block 306 can be implemented by usingface tracking algorithms or analytics (i.e. a routine or set ofinstructions for tracking the facial plane orientation of a user's face)to approximate the point of regard 8. The concept of face tracking iswell known and a number of different techniques have been developed foraccomplishing automatic face tracking.

At block 308, a display controller instruction is selected based on thepoint of regard 8 that was determined at block 306. In the exampleembodiment shown, if the point of regard 8 is outside the viewing area13 of the display 12, the display controller 60 selects an “off”instruction. If the point of regard 8 is within the viewing area 13, thedisplay controller 60 selects an “on” instruction. It can be appreciatedthat additional types of instructions can be selected such as dimmingthe display 12, adjusting size or focus of the content (e.g. blurring),locking out the device, or activating a screensaver, to name a few.

At block 310, the display controller instruction which has been selectedis sent to display 110. As shown by the dotted line 312, the blocks 300to 310 may be repeated to track a user 2 over a period of time. Thefrequency of tracking is determined at block 302 as part of determininga time to track the user 2. In another embodiment, instructions can alsobe sent to components of the mobile device 100 other than the display12. For example, rather than or in addition to controlling the display,block 310 could instruct the subsystem responsible for security (notshown) to password-lock the mobile device 100.

In the above examples, the display controller application 60 detects andtracks the presence or absence of a single user 2. In other embodiments,the application 60 may be used to control the display 12 according tothe detection of one or more users 2 and their respective points ofregard 8. In such embodiments, it can be appreciated that the displaycontroller application 60 may act to reduce visibility of what is beingdisplayed even if, or especially because, one or more subjects arecaptured in the image. For example, when reviewing sensitive content,the display controller application 60 can trigger an instruction toreduce the visibility of the content when a second subject is detectedin the image, for security purposes.

FIG. 10 shows a flow diagram of an example PCCR process to determine thepoint of regard based on eye tracking using the mobile device 100. Itcan be appreciated that the following description of the eye trackingoperations is for illustrative purposes and that other methods of eyetracking are equally applicable to the principles discussed herein. Atblock 400, the user is illuminated by the light source 32. At block 402,an image of the user 2 illuminated by the light source 32 is captured bythe front camera lens 30. As discussed in FIG. 8, blocks 400 and 402 arecarried out at block 304.

At block 404 and 406, the center of the pupil and the center of theglint are identified, respectively. The relative vector between thecenter of the pupil and the center of the glint is then determined atblock 408.

At block 410, the distance between the front camera lens 30 and theuser's eyes 4 in the image is measured. The devices and methods ofdetermining the distance between an object and a camera are known andcan be used here. Examples of distance sensing devices include a camera,infrared optics, and an ultrasonic transmitter and receiver. Differentcombinations of distance sensing devices can also be used, and knownmethods associated with the devices for finding the distance between anobject and a camera device 31 are applicable. In the example embodimentof FIG. 2, distance from the front camera lens 30 to an object ismeasured by a distance sensor 34 placed adjacent to the front cameralens 30. Although the distance sensor 34 is illustrated as a separatecomponent in FIG. 2, the distance sensor 34 can be incorporated intoother components of the mobile device 100. For example, a distancesensor based on infrared light can be incorporated into an infraredlight source 32 also used to illuminate the user 2 when an image iscaptured.

At block 412, the point of regard 8 of the user 2 is calculated based onknown methods using the relative vector and the distance between thefront camera lens 30 and the user's eyes 4.

FIG. 11 shows a flow diagram of an example face tracking process toapproximate the point of regard using the mobile device 100. It can beappreciated that the following description of the face-trackingoperations is for illustrative purposes and that another method of facetracking is equally applicable to the mobile device 100. At block 500,the user 2 is illuminated by the light source 32 of the mobile device100. At block 502, an image of the user 2 illuminated by the lightsource 32 is captured by the front camera lens 30. As discussed in FIG.8, blocks 500 and 502 are carried out at block 304.

At block 504, the face of the user 2 is detected and at block 506, a setof non-planar reference points of the face (e.g. facial features such aseyes, mouth and nose) is detected. Facial recognition algorithms areknown and can be used here.

At block 508, the orientation of the set of non-planar reference pointsis used to extrapolate the user's facial plane orientation. A vectornormal to the facial plane is then generated at block 510. The normalvector of the facial plane is used to approximate the gaze direction 6of the user 2.

At block 512, the vector between the user's eyes and the front cameralens 30 is generated to compare with the normal vector of the user'sfacial plane.

At block 514, the point of regard 8 of the user 2 is approximated byevaluating the alignment of the normal vector of the facial plane andthe vector between the user's eyes 4 and the front camera lens 30. Thepoint of regard 8 of the user 2 will lie within the viewing area 13 ifthe dot products of the two vectors are sufficiently aligned, wherealignment may be evaluated using a user-specified or pre-programmedthreshold. If the calculated dot product exceeds the threshold, thepoint of regard 8 is considered to lie outside the viewing area 13.

Turning to FIG. 12, a schematic diagram is provided showing a first user2 and a second user 2′ viewing the display 12 of the mobile device 100.In this example, the mobile device 100 is situated in front of the users2, 2′. The users 2,2′ have respective pairs of eyes 4, 4′ that haveassociated therewith, respective gaze directions 6, 6′, and respectivepoints of regard 8, 8′. The front camera lens 30, light source 32, anddistance sensor 34 (collectively referred to as the camera device 31),can be used to track the gaze directions 6, 6′ and points of regard 8,8′of the respective users 2, 2′.

In an example embodiment, the point of regard module 212 can be modifiedto be capable of detecting one or more users 2 and to determine theirrespective points of regard 8.

Example computer executable instructions for controlling the display 12of the mobile device 100 based on eye tracking of one or more users 2are provided in FIG. 13. At block 300, status information on the mobiledevice 100 is received or otherwise obtained. Based on the statusinformation, a time to track one or more users 2 of the mobile device100 is determined at block 302. At block 304, an image of all users 2that happen to be in the field of view of the camera lens 30 of themobile device 100 is captured using the front camera lens 30. Using theimage captured in block 304, the points of regard 8 of one or more users2 is determined at block 606. At block 608, a display controllerinstruction is selected based on the point of regard 8 of one or moreusers 2. At block 310, the display controller instruction which has beenselected is sent to display 110. As shown by the dotted line 312, theblocks 300 to 310 may be repeated to track one or more users 2 over aperiod of time. The frequency of tracking is determined at block 302 aspart of determining a time to track the user 2.

As noted above, at block 606, the point of regard 8 of one or more users2 is determined. In an example embodiment, block 606 can be implementedby repeating the eye tracking process shown in FIG. 10 (for determiningthe point of regard of a single user 2) for each user 2 captured in theimage. In another example embodiment, block 606 can be implemented byrepeating the face tracking process shown in FIG. 11 (for approximatingthe point of regard of a single user 2) for each user 2 captured in theimage.

As noted above, at block 608, a display controller instruction isselected based on the points of regard 8 of one or more users 2. In anexample embodiment, if more than one point of regard 8 is detected tolie within the viewing area 13, the display controller 60 can select an“off” instruction in order to prevent more than one user 2 from viewingthe display 12 at the same time, e.g. for security reasons. In anotherexample embodiment, the same can be achieved by selecting an “off”instruction if more than one user 2 is detected. Similar logic appliesto other control instructions such as entering a “lock-out” mode ordimming the display 12.

Blocks 300, 302, 304 and 310 as discussed in the display controller 60for a single user 2 are equally applicable for more than one user 2 andcan be used here.

In another example embodiment, the display controller 60 can select aninstruction based on the point of regard 8 of one specific user 2 out ofthe set of users 2 captured in the image. For example, the displaycontroller 60 can select an instruction based on the user 2 at theshortest distance from the mobile device 100, as measured or otherwiseobtained by the distance sensor 34. Such a user 2 is likely to be theuser 2 operating the mobile device 100 (referred to as the “operatinguser”). By tracking the operating user only, the display controller 60can function in a similar manner as the single user 2 case.

In another example embodiment, the display controller 60 can select aninstruction based on the point of regard 8 of a subset of specific users2 out of the set of users captured in the image. For example, thedisplay controller 60 can select an “off” instruction if any user 2,that is not the operating user, has a point of regard 8 within theviewing area 13. In this example, the display controller 60 preventsanother user 2 from spying or inadvertently observing content on thedisplay 12 while the mobile device 100 is used by the operating user,again for security purposes.

As discussed above, the calibration module 211 can be used to initiate atraining routine to train the display controller application 60. FIG. 14illustrates an example set of computer executable instructions forimplementing an example training routine. At 700, the calibration module211 detects initiation of the calibration routine. For example, themobile device 100 may perform periodic calibrations, a user may selectan option from a menu (not shown), or an application may make such arequest, e.g. based on quality factors or other criteria. At 702, thecalibration module 211 enables a prompt to be displayed (not shown) thatinstructs the user to move the mobile device 100 into differentpositions with respect to their face/eyes. For example, the prompt maystep through a series of positions asking the user 2 to hold thepositions until an image has been taken. Any number of distinctpositions can be used. At 704, an image is captured at each position.From the image, various characteristics can be determined such as whatshape the face/eyes take when viewed from the particular positioncorresponding to that image. The accelerometer 125 may also have aparticular reading that can be correlated with the image characteristicsat 706. The calibration data may then be stored at 708. The calibrationdata can be used to enable the display controller application 60 tooperate more efficiently. For example, if the display controllerapplication 60 would obtain a false negative due to the subject taking aparticular shape in a particular position but during training it isdetermined that when the user tilts the mobile device 100 in aparticular way they appear in the form of the particular shape when animage is taken, such false negatives can be avoided. As noted above,during the calibration process, the display controller application 60can be operable to prompt the user to both look straight at the display12 for a particular held position, and then look away from the mobiledevice 10 at the same position. In this way, the display controllerapplication 60 is provided with two points of reference to thereafterjudge whether or not the user is likely glancing at the display 12.Accordingly, operation 704 in FIG. 14 may include the acquisition ofmore than one image at each position.

It will be appreciated that the examples and corresponding diagrams usedherein are for illustrative purposes only. Different configurations andterminology can be used without departing from the principles expressedherein. For instance, components and modules can be added, deleted,modified, or arranged with differing connections without departing fromthese principles.

The steps or operations in the flow charts and diagrams described hereinare just for example. There may be many variations to these steps oroperations without departing from the spirit of the invention orinventions. For instance, the steps may be performed in a differingorder, or steps may be added, deleted, or modified.

Although the above has been described with reference to certain specificexample embodiments, various modifications thereof will be apparent tothose skilled in the art as outlined in the appended claims.

1. A method of controlling a display of a mobile device including acamera, comprising: capturing a first image including a subject, usingthe camera of the mobile device, the camera being directed in a samedirection as the display; determining in the first image, a first pointof regard indicative of an area on the display at which a gaze of thesubject is directed; capturing a second image using the camera accordingto a frequency; determining a second point of regard in the secondimage; and reducing or increasing visibility of at least one portion ofwhat is displayed on the display when the second point of regard isdirected to an area off the display.
 2. The method of claim 1, whereinthe frequency is determined according to at least one predeterminedcriterion.
 3. The method of claim 2, further comprising determining theat least one predetermined criterion, and capturing the first imageafter meeting the at least one criterion.
 4. The method of claim 3,wherein the at least one criterion is associated with any one or more ofthe following: a period time, a location associated with the mobiledevice, content being displayed on the display, presence of a pluralityof subjects in a field of view of the camera, a mode of the mobiledevice, a change in orientation of the mobile device, an interactivitywith one or more input devices, and one or more applications currentlybeing used.
 5. The method of claim 1, wherein the reducing or increasingvisibility corresponds to a manner in which to modify what is viewed onthe display.
 6. The method of claim 5, wherein the display is modifiedby performing any one or more of the following: turning off the one ormore portions of the display, initiating a screensaver, dimming abacklight, adjusting a focus of content being displayed, and locking-outthe device.
 7. The method of claim 1, wherein the first point of regardand second point of regard are determined using at least one of eyetracking and face tracking.
 8. The method of claim 1, wherein thefrequency is determined according to at least one of: a sensitivity ofcontent displayed on the display, a type of content displayed, anapplication running on the mobile device, and a mode of the mobiledevice.
 9. The method of claim 8, wherein the frequency is proportionalto the sensitivity of the content.
 10. The method of claim 8, whereinthe mode of the mobile device is a security mode.
 11. A mobile devicecomprising a processor, memory, a display, and a camera, the memorystoring computer executable instructions for: capturing a first imageincluding a subject, using the camera of the mobile device, the camerabeing directed in a same direction as the display; determining in thefirst image, a first point of regard indicative of an area on thedisplay at which a gaze of the subject is directed; capturing a secondimage using the camera according to a frequency; determining a secondpoint of regard in the second image; and reducing or increasingvisibility of at least one portion of what is displayed on the displaywhen the second point of regard is directed to an area off the display.12. A non-transitory computer readable storage medium for controlling adisplay of a mobile device including a camera, comprising computerexecutable instructions for: capturing a first image including asubject, using the camera of the mobile device, the camera beingdirected in a same direction as the display; determining in the firstimage, a first point of regard indicative of an area on the display atwhich a gaze of the subject is directed; capturing a second image usingthe camera according to a frequency; determining a second point ofregard in the second image; and reducing or increasing visibility of atleast one portion of what is displayed on the display when the secondpoint of regard is directed to an area off the display.
 13. Thenon-transitory computer readable storage medium of claim 12, wherein thefrequency is determined according to at least one predeterminedcriterion.
 14. The non-transitory computer readable storage medium ofclaim 13, further comprising instructions for determining the at leastone predetermined criterion, and capturing the first image after meetingthe at least one criterion.
 15. The non-transitory computer readablestorage medium of claim 14, wherein the at least one criterion isassociated with any one or more of the following: a period time, alocation associated with the mobile device, content being displayed onthe display, presence of a plurality of subjects in a field of view ofthe camera, a mode of the mobile device, a change in orientation of themobile device, an interactivity with one or more input devices, and oneor more applications currently being used.
 16. The non-transitorycomputer readable storage medium of claim 12, wherein the reducing orincreasing visibility corresponds to a manner in which to modify what isviewed on the display.
 17. The non-transitory computer readable storagemedium of claim 16, wherein the display is modified by performing anyone or more of the following: turning off the one or more portions ofthe display, initiating a screensaver, dimming a backlight, adjusting afocus of content being displayed, and locking-out the device.
 18. Thenon-transitory computer readable storage medium of claim 12, wherein thefirst point of regard and second point of regard are determined using atleast one of eye tracking and face tracking.
 19. The non-transitorycomputer readable storage medium of claim 12, wherein the frequency isdetermined according to at least one of: a sensitivity of contentdisplayed on the display, a type of content displayed, an applicationrunning on the mobile device, and a mode of the mobile device.
 20. Thenon-transitory computer readable storage medium of claim 19, wherein thefrequency is proportional to the sensitivity of the content.
 21. Thenon-transitory computer readable storage medium of claim 19, wherein themode of the mobile device is a security mode.